Systems and methods for calibrating a machine learning model

ABSTRACT

Systems and methods include: collecting digital threat scores of an incumbent digital threat machine learning model; identifying incumbent and successor digital threat score distributions; identifying quantiles data of the incumbent digital threat score distribution; collecting digital threat scores of a successor digital threat machine learning model; calibrating the digital threat scores of the successor digital threat score distribution based on the quantiles data of the incumbent digital threat score distribution and the incumbent digital threat score distribution; and in response to remapping the digital threat scores of the successor digital threat score distribution, publishing the successor digital scores in lieu of the incumbent digital threat scores based on requests for digital threat scores.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/533,564, filed on 17 Jul. 2017, U.S. Provisional Application Ser.No. 62/543,952, filed on 10 Aug. 2017, which are incorporated in theirentireties by this reference.

TECHNICAL FIELD

This invention relates generally to the digital fraud and abuse field,and more specifically to a new and useful system and method fordetecting digital fraud or digital abuse and evolving underlying machinelearning models in the digital fraud and abuse field.

BACKGROUND

The modern web and Internet enables entities to engage and perform anincalculable amount of activities. Many of these activities involveuser-to-user activities, user-to-business activities (or the reverse),and the like. These activities between users and between users andorganizational entities over the web often involve the access, use,and/or exchange of information by one or more of the parties of theactivities. Because of the malleable nature of the digital realm thatthese activities operate within, there arise a countless number ofdigital threats by digital actors that aim to commit digital fraudand/or digital abuse using online services and/or Internet-accessibleapplications (e.g., web or mobile applications). Additionally, some ofthese bad digital actors may also aim to misappropriate the information(e.g., hack) being exchanged between legitimate entities to theseactivities. These digital threats may also be perpetrated by maliciousthird-parties who seek to unlawfully or otherwise, impermissibly takeadvantage of the data or information that is exchanged or, if notexchanged, data or information about the activities or actions of usersand/or businesses on the web.

Other digital threats involving a malicious party or a bad digital actorthat acts unilaterally (or in concert with other malicious actors) toabuse digital resources of a service provider to perpetrate fraud orother unlawful activities that are also of significant concern tolegitimate service providers and users of the Internet.

While there may currently exist some technologies that attempt to detectdigital fraud and digital abuse or other malicious digital activitiesover the Internet, these existing technology implementations may notsufficiently detect malicious digital activities over the Internet withaccuracy and in real-time to provide an opportunity for an appropriateresponse by an affected party. Additionally, these existing technologyimplementations lack the capabilities to detect new and/or never beenencountered before digital threats and automatically (or nearautomatically) evolve the technology implementation to effectivelyrespond and neutralize the digital threats.

Therefore, there is a need in the digital fraud and abuse field for adigital fraud and abuse solution that enables effective detection ofmultiple and specific digital threats involving digital fraud and/ordigital abuse via digital resources of a service provider. Theembodiments of the present application described herein providetechnical solutions that address, at least, the need described above.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system in accordancewith one or more embodiments of the present application;

FIG. 2 illustrates an example method in accordance with one or moreembodiments of the present application;

FIG. 3 illustrates an example schematic of a digital threat mitigationplatform in accordance with one or more embodiments of the presentapplication;

FIG. 4 illustrates an example schematic of a digital threat mitigationplatform in accordance with one or more embodiments of the presentapplication;

FIG. 5 illustrates an example schematic of a digital threat mitigationplatform in accordance with one or more embodiments of the presentapplication;

FIG. 6 illustrates an example schematic of a digital threat mitigationplatform in accordance with one or more embodiments of the presentapplication; and

FIG. 7 illustrates an example schematic of a sub-system for calibratingmodel changes in accordance with one or more embodiments of the presentapplication.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the presentapplication are not intended to limit the inventions to these preferredembodiments, but rather to enable any person skilled in the art to makeand use these inventions.

1. Overview

As discussed above, digital threats are abounding and continue to evolveto circumvent existing digital fraud detection technologies. Theevolving nature of digital threats compounded with the great number oftransactions, events, actions, and/or activities (exceeding billions innumber) occurring over the web and/or Internet highlight the manydeficiencies of traditional digital fraud detection and threatmitigation implementations.

The embodiments of the present application, however, provide an advancedtechnology platform that is capable of ingesting billions of digitalevents and/or transactions over the Internet, the web, web applications,mobile applications, and the like and dynamically implement digitalthreat mitigation implementations that are capable of detectingmalicious activities, fraudulent activities, digital abuses and generatedigital threat mitigation recommendations and responses that operate tomitigate and/or eliminate the digital fraud and abuse threats stemmingfrom the malicious or fraudulent activities.

The advanced technology platform of many embodiments of the presentapplication employs a robust ensemble of machine learning models andrelated systems that operate to ingest the great number of digitalactivities performed and events occurring over the web. Accordingly,using these finely tuned and perpetually evolving and tunable machinelearning models, a system implementing the several embodiments of thepresent application can predict a threat level and/or classify a digitalthreat with high accuracy and, in some embodiments, in real-time (e.g.,as the event is occurring or shortly thereafter) compute a digitalthreat score for each event or activity that is received by the system.

The digital threat score may be exposed via a score application programinterface (API) that may function to interact with various endpoints ofthe digital threat mitigation platform. Specifically, the score API mayfunction to interact with one or more computing servers that implementthe ensembles of machine learning models used to predict a likelihood ofdigital fraud and/or digital abuse. The score API may function to returna value (e.g., a number, likelihood or probability, or other criterion)that indicates how likely it is that an actor involved or associatedwith digital events and/or activities is a malicious actor or may beperpetrating cyber fraud or digital abuse (e.g., payment abuse, etc.).Accordingly, the digital threat score calculated by the score API may beused in several manners including to inform digital event dataprocessing decisions (e.g., deny, hold, or approve digital transaction)or to define which of one or more digital threat mitigation protocols orimplementations that should be applied to future digital event dataand/or current the digital events to mitigate or eliminate a digitalthreat associated therewith.

Additionally, recognizing that in some circumstances service providersthat provide online and/or digital resources to users may need tomitigate or prevent multiple forms of digital fraud and/or digital abusesimultaneously, the embodiments of the present application enable thegeneration of a global digital threat score and a plurality of specificdigital threat scores for varying, known digital fraud or abuse types.Accordingly, while the global digital threat score may indicate to theservice provider a general indication of the existence of digital fraudand/or digital abuse in digital events data, the specific digital threatscores for the plurality of digital abuse types function to specificallyidentify a type of fraud or abuse that is being committed in a digitalevents dataset. These specific indications allow the service providerand/or the digital threat mitigation platform to implement digitalthreat mitigation or prevention actions that effectively address thespecific digital fraud or abuse.

2. Model Calibration Overview

As digital threats evolve, it is also necessary that the digital threatmitigation platform evolve the underlying digital threat detectingmachine learning models that generally function to identify digitalevents that may involve a level of digital threat or digital abuse.However, changing the underlying digital threat machine learning modelsmay adversely affect the distribution of threats detected by theplatform and/or service providers using the digital threat mitigationplatform as the thresholds implemented by service providers fordetecting digital threats and non-digital threats may remain constant.

Accordingly, a change in an underlying digital threat machine learningmodel implemented digital event classification for a specific serviceprovider using the digital threat detection services of the digitalthreat mitigation platform may cause unexpected shifts or unusuallylarge shifts in the classification proportions of users of the servicesof the service provider or the classification proportions of eventsinvolving the services of the service provider that the machine learningmodel scores or predicts to be or involve a threat. That is, the digitalthreat scores generated by a new or a target successor model that isreplacing an existing or incumbent model of a specific service providermay produce digital threat scores with a different score distributionthan a prior score distribution of the incumbent model and in thecontext of using fixed evaluation thresholds for initiating automatedreactions (e.g., blocking an order, banning a user, etc.) to events orusers, this may cause large shifts in the proportion of events or usersthat receive the automated reactions. Accordingly, this sensitivity tothe shifts of a number of automated reactions and/or non-automatedreactions (e.g., manual reviews) that is experienced by serviceproviders is primarily a result of a lack of an update to the fixedautomated decisioning thresholds even though the underlying machinelearning scoring model has been changed.

In some embodiments, the sensitivity level that may be experienced by aservice provider due to a migration from an old machine learning modelto a new machine learning model may be measured or estimated. Themeasured sensitivity may be used the machine learning system of thepresent application to trigger a calibration of a new machine learningmodel being installed for a specific service provider. For instance, anumber of automated decision for a given dataset may be calculated foreach of an old and new machine learning model. If the machine learningsystem measures that an increase or decrease of a number of automateddecisions has increased or decreased based on the scoring outcomes ofthe new machine learning model at or beyond one or more sensitivitythresholds, the machine learning system may determine that thesensitivity level experienced by a service provider may be significantand therefore, trigger a calibration of a proposed new machine learningmodel prior to enabling live scoring with the new machine learningmodel.

Thus, while the new digital threat scoring model may produce new andmore accurate digital threat scores, the predetermined or fixedautomated decisioning thresholds of a specific service provider may nothave been updated or changed to take into account the changes in anunderlying threat score distribution of the new digital threat scoringmachine learning model. As a result, significant disruptions in theability of a service provider or the platform to accurately performdigital threat detection may be caused because too few or too largedigital threat detections in specific classification classes may beidentified as a result of the new digital threat scores of the newdigital threat model not being in line with the prior or maintaineddigital threat detection thresholds (i.e., the automated decisioningthresholds). It shall be noted that automated decisioning thresholds forspecific services providers may not necessarily be known to a machinelearning service and/or the digital threat mitigation platform thatperforms one or more of the methods described herein. Nevertheless, themachine learning service or other system implementing the methods mayfunction to calibrate successor machine learning models without anexpress knowledge of the automated decisioning thresholds.

Additionally, because the digital threat mitigation platform mayfunction to produce many model changes and updates (e.g., hundreds orthousands of model changes and updates), it may be extremely difficultto continually determine proper updates to the thresholds used fordetecting digital threat based on the digital threat scores produced bythe new digital threat scoring models.

The embodiments of the present application, however, provide systems andmethods that enable the evolution and changes to an underlying machinelearning model for detecting digital threats while minimizing orameliorating the probabilities of disruption in the detection of digitalthreat events using prior or unchanged digital threat detectionthresholds applied against new digital threat scores of new digitalthreat models being implemented in the underlying machine learning modelchange. In this way, while a service provider or the like may bemigrated to an improved digital threat scoring machine learning modelthat may detect digital threats with higher accuracy and faster than aprior model, confusing disruptions in the digital threat detectioncapabilities are mitigated by preserving a digital threat scoredistribution between a prior digital threat scoring machine learningmodel and the new digital threat scoring machine learning model.

3. System for Digital Fraud and/or Abuse Detection and Scoring

As shown in FIG. 1, a system 100 for detecting digital fraud and/ordigital abuse includes one or more digital event data sources 110, a webinterface 120, a digital threat mitigation platform 130, and a serviceprovider system 140.

The system 100 functions to enable a prediction of multiple types ofdigital abuse and/or digital fraud within a single stream of digitalevent data. The system 100 provides web interface 120 that enables usersto generate a request for a global digital threat score andadditionally, make a request for specific digital threat scores forvarying digital abuse types. After or contemporaneously with receiving arequest from the web interface 120, the system 100 may function tocollect digital event data from the one or more digital event datasources no. The system 100 using the digital threat mitigation platform130 functions to generate a global digital threat score and one or morespecific digital threat scores for one or more digital abuse types thatmay exist in the collected digital event data.

The one or more digital event data sources no function as sources ofdigital events data and digital activities data, occurring fully or inpart over the Internet, the web, mobile applications, and the like. Theone or more digital event data sources 110 may include a plurality ofweb servers and/or one or more data repositories associated with aplurality of service providers. Accordingly, the one or more digitalevent data sources 110 may also include the service provider system 140.

The one or more digital event data sources 110 function to captureand/or record any digital activities and/or digital events occurringover the Internet, web, mobile applications (or other digital/Internetplatforms) involving the web servers of the service providers and/orother digital resources (e.g., web pages, web transaction platforms,Internet-accessible data sources, web applications, etc.) of the serviceproviders. The digital events data and digital activities data collectedby the one or more digital event data sources 110 may function as inputdata sources for a machine learning system 132 of the digital threatmitigation platform 130.

The digital threat mitigation platform 130 functions as an engine thatimplement at least a machine learning system 132 and, in someembodiments, together with a warping system 133 to generate a globalthreat score and one or more specific digital threat scores for one ormore digital abuse types. The digital threat mitigation platform 130functions to interact with the web interface 120 to receive instructionsand/or a digital request for predicting likelihoods of digital fraudand/or digital abuse within a provided dataset. The digital threatmitigation engine 130 may be implemented via one or more specificallyconfigured web or private computing servers (or a distributed computingsystem) or any suitable system for implementing system 100 and/or method200.

The machine learning system 132 functions to identify or classifyfeatures of the collected digital events data and digital activity datareceived from the one or more digital event data sources no. The machinelearning system 132 may be implemented by a plurality of computingservers (e.g., a combination of web servers and private servers) thatimplement one or more ensembles of machine learning models. The ensembleof machine learning models may include hundreds and/or thousands ofmachine learning models that work together to classify features ofdigital events data and namely, to classify or detect features that mayindicate a possibility of fraud and/or abuse. The machine learningsystem 132 may additionally utilize the input from the one or moredigital event data sources 110 and various other data sources (e.g.,outputs of system 100, system 100 derived knowledge data, externalentity-maintained data, etc.) to continuously improve or accurately tuneweightings associated with features of the one or more of the machinelearning models defining the ensembles.

The warping system 133 of the digital threat mitigation platform 130, insome embodiments, functions to warp a global digital threat scoregenerated by a primary machine learning ensemble to generate one or morespecific digital threat scores for one or more of the plurality ofdigital abuse types. In some embodiments, the warping system 133 mayfunction to warp the primary machine learning ensemble, itself, toproduce a secondary (or derivative) machine learning ensemble thatfunctions to generate specific digital threat scores for the digitalabuse and/or digital fraud types. Additionally, or alternatively, thewarping system 133 may function to implement a companion machinelearning model or a machine learning model that is assistive indetermining whether a specific digital threat score should be generatedfor a subject digital events dataset being evaluated at the primarymachine learning model. Additionally, or alternatively, the warpingsystem 133 may function to implement a plurality of secondary machinelearning models defining a second ensemble that may be used toselectively determine or generate specific digital threat scores.Accordingly, the warping system 133 may be implemented in variousmanners including in various combinations of the embodiments describedabove.

The digital threat mitigation database 134 includes one or more datarepositories that function to store historical digital event data. Thedigital threat mitigation database 134 may be in operable communicationwith one or both of an events API and the machine learning system 132.For instance, the machine learning system 132 when generating globaldigital threat scores and specific digital threat scores for one or morespecific digital abuse types may pull additional data from the digitalthreat mitigation database 134 that may be assistive in generating thedigital threat scores.

The ensembles of machine learning models may employ any suitable machinelearning including one or more of: supervised learning (e.g., usinglogistic regression, using back propagation neural networks, usingrandom forests, decision trees, etc.), unsupervised learning (e.g.,using an Apriori algorithm, using K-means clustering), semi-supervisedlearning, reinforcement learning (e.g., using a Q-learning algorithm,using temporal difference learning), and any other suitable learningstyle. Each module of the plurality can implement any one or more of: aregression algorithm (e.g., ordinary least squares, logistic regression,stepwise regression, multivariate adaptive regression splines, locallyestimated scatterplot smoothing, etc.), an instance-based method (e.g.,k-nearest neighbor, learning vector quantization, self-organizing map,etc.), a regularization method (e.g., ridge regression, least absoluteshrinkage and selection operator, elastic net, etc.), a decision treelearning method (e.g., classification and regression tree, iterativedichotomiser 3, C4.5, chi-squared automatic interaction detection,decision stump, random forest, multivariate adaptive regression splines,gradient boosting machines, etc.), a Bayesian method (e.g., naïve Bayes,averaged one-dependence estimators, Bayesian belief network, etc.), akernel method (e.g., a support vector machine, a radial basis function,a linear discriminate analysis, etc.), a clustering method (e.g.,k-means clustering, expectation maximization, etc.), an associated rulelearning algorithm (e.g., an Apriori algorithm, an Eclat algorithm,etc.), an artificial neural network model (e.g., a Perceptron method, aback-propagation method, a Hopfield network method, a self-organizingmap method, a learning vector quantization method, etc.), a deeplearning algorithm (e.g., a restricted Boltzmann machine, a deep beliefnetwork method, a convolution network method, a stacked auto-encodermethod, etc.), a dimensionality reduction method (e.g., principalcomponent analysis, partial lest squares regression, Sammon mapping,multidimensional scaling, projection pursuit, etc.), an ensemble method(e.g., boosting, boostrapped aggregation, AdaBoost, stackedgeneralization, gradient boosting machine method, random forest method,etc.), and any suitable form of machine learning algorithm. Eachprocessing portion of the system 100 can additionally or alternativelyleverage: a probabilistic module, heuristic module, deterministicmodule, or any other suitable module leveraging any other suitablecomputation method, machine learning method or combination thereof.However, any suitable machine learning approach can otherwise beincorporated in the system 100. Further, any suitable model (e.g.,machine learning, non-machine learning, etc.) can be used in generatingendpoint health intelligence and/or other data relevant to the system100.

The service provider 140 functions to provide digital events data to theone or more digital event data processing components of the system 100.Preferably, the service provider 140 provides digital events data to anevents application program interface (API) associated with the digitalthreat mitigation platform 130. The service provider 140 may be anyentity or organization having a digital or online presence that enableusers of the digital resources associated with the service provider'sonline presence to perform transactions, exchanges of data, perform oneor more digital activities, and the like.

The service provider 140 may include one or more web or privatecomputing servers and/or web or private computing devices. Preferably,the service provider 140 includes one or more client devices functioningto operate the web interface 120 to interact with and/or communicationwith the digital threat mitigation engine 130.

The web interface 120 functions to enable a client system or clientdevice to operably interact with the remote digital threat mitigationplatform 130 of the present application. The web interface 120 mayinclude any suitable graphical frontend that can be accessed via a webbrowser using a computing device. The web interface 120 may function toprovide an interface to provide requests to be used as inputs into thedigital threat mitigation platform 130 for generating global digitalthreat scores and additionally, specific digital threat scores for oneor more digital abuse types. In some embodiments, the web interface 120includes an application program interface that is in operablecommunication with one or more of the computing servers or computingcomponents of the digital threat mitigation platform 130.

The web interface 120 may be used by an entity or service provider tomake any suitable request including requests to generate global digitalthreat scores and specific digital threat scores.

Additionally, as shown in FIG. 2-FIG. 6, the systems and methodsdescribed herein may implement the digital threat mitigation platform inaccordance with the one or more embodiments described in the presentapplication as well as in the one or more embodiments described in U.S.patent application Ser. No. 15/653,373, which is incorporated byreference in its entirety.

3.1 Sub-System for Calibrating for ML Model Changes

As shown in FIG. 7, a sub-system 700 of the system 100 for calibratingfor digital threat score model changes include a first reservoir 710, asecond reservoir 720, and a third reservoir 730.

The first reservoir 710 may include a database or datastore that ispreferably configured to store samples of digital threat scoresgenerated by a target digital threat score machine learning model, suchas an incumbent digital threat score machine learning model as describedherein below. The first reservoir 710 may include logic or softwaremodules, such as a counter module and quantiles capturing module, thatfunction to generate a count value for each of the digital threat scorescollected and stored by the first reservoir 710 and further, capture oridentify quantiles of a score distribution formed by the collecteddigital threat scores of the target model.

The first reservoir 710 may also function to collect raw or uncalibrateddigital threat scores of a second digital threat score machine learningmodel and similarly, increment a separate counter for each of thedigital threat scores collected therefrom and further, produce quantilesdata thereof.

The first reservoir 710 may function to output the quantiles data and,in some embodiments, the score distribution data of each of the targetmodel and the successor model to a second reservoir 720.

The second reservoir 720 may primarily function as a quantiles reservoirthat stores quantiles data for each of the target model and thesuccessor model, preferably during calibration of the raw scores of thesuccessor model. The second reservoir 720 may function to store multiplequantiles for each of the target model and successor model andpreferably, together with a timestamp of creation of the quantiles sothat a remapping function may be able to identify quantiles capturedlatest in time.

The second reservoir 720 may function to output to a third reservoir 730quantiles data of a target model such that a remapping function of thethird reservoir 730 may function to implement a remapping of the raw oruncalibrated scores of a successor digital threat ML model.

The third reservoir 730 may function as a remapping reservoir thatfunctions to calibrate the uncalibrated scores of a successor digitalthreat machine learning model by remapping uncalibrated scores of thesuccessor digital threat model to the digital threat scores of thetarget model. As described in more detail herein below, the remappingfunction implemented by the third reservoir 730 may be triggered basedon the satisfaction of one or more conditions within the secondreservoir 720 and/or the first reservoir 710 relating, at least, to arecording of score distributions of the target model and successor modeland recordation of quantiles of the score distributions of the targetmodel. Once triggered, the remapping function of the third reservoir mayfunction to remap digital threat scores between a pair of scoredistributions including the score distribution of the target model andthe score distribution of the uncalibrated successor model. Using amonotonic function, the remapping function may generate a calibratedscore distribution for the successor model that maintains or isapproximate to the score distribution of the target model.

The third reservoir 730 may function to record any remapping of theuncalibrated score distributions of the successor model as well as thecalibrated score distributions derived therefrom. Additionally, thethird reservoir 370 may automatically publish calibrated digital threatscores of the successor model based on a detection or completion of theremapped or calibrated score distribution of the successor model.Additionally, or alternatively, the automatic publishing of thecalibrated digital threat scores of the successor model may beautomatically triggered based on an automatic comparison of the scoredistribution of target model and the calibrated score distribution ofthe successor model having a variance below a maximum distributionvariance threshold. The maximum distribution variance threshold may be apredetermined value or a dynamic value determined based at least on acount of digital threat score values that make up the scoredistributions for the target and successor models. The lower the count,the greater the dynamic variance threshold may be and the higher thecount of digital threat score, the lower the dynamic variance thresholdmay be.

Alternatively, the third reservoir may function to reserve thecalibrated digital threat scores of the calibrated score distribution ofthe successor model and publish calibrated digital threat scores onlyafter a verification of the calibrated score distribution is performed.

4. Method for Mitigated Model Reconfiguration and Model ScoreCalibration

As shown in FIG. 2, the method 200 includes collecting digital threatscores of an incumbent machine learning model S210, generating a digitalthreat score distribution for the incumbent machine learning model S220,identifying quantiles for the digital threat score distribution for theincumbent S225, collecting digital threat scores of a successor machinelearning model S230, generating a digital threat score distribution forthe successor machine learning model S240, remapping the digital threatscore of the successor distribution machine learning model S250, andreturning one or more digital threat scores S260.

The method 200 preferably functions to enable a seamless migration froma prior machine learning model (i.e., incumbent ML model) to a newmachine learning model (i.e., successor ML model) with improvedclassification accuracy and/or improved predictive capabilities. In apreferred embodiment, the method 200 is implemented in a live and/oronline (not offline) system that is actively connected to the Internet(or one or more web servers, etc.) and performs digital threat detectionbased on live digital events, live threat scoring requests, and thelike. In such preferred embodiments, while a prior or an incumbent MLmodel is used to ingest digital events data and produce live threatscores, a second (raw/uncalibrated) new or successor ML model may beimplemented simultaneously in a shadow mode during a period in which thesame digital events data is ingested and digital threat scores producedby the successor ML model that are not returned or exposed for thedigital events data under evaluation.

Accordingly, the method 200 includes capturing a threat scoredistribution of the incumbent ML model as well as a threat scoredistribution of the successor ML model based on a same set oftimestamped digital events data. While the threat score distributions ofthe incumbent ML model and the successor ML model may vary, the method200 functions to (approximately) preserve the general threat scoredistribution of the incumbent ML model within the threat scoredistribution of successor ML model by capturing score quantiles data ofthe threat score distributions of the incumbent ML model and thesuccessor ML model and using the quantiles data to produce a monotonicfunction. The monotonic function may be applied to the threat scoredistribution of the successor ML model to remap the raw scores thereofsuch that the threat score distribution of the incumbent ML model is(approximately) preserved.

Once this initial quantiles-driven calibration is imposed on the threatscore distribution of the successor ML model, the method 200additionally, or alternatively, may function to perform classificationaccuracy testing and calibration between the raw threat scores and thequantiles-driven calibrated threat scores of the successor ML model maybe performed. Specifically, the method 200 may optionally derive ROCcurve data for each of the raw/uncalibrated successor ML model and thecalibrated successor ML model based on a same set of digital events data(e.g., live digital events or historical digital events data). Once theROC curve data is determined, the method 200 may function to compare theROC curve for the uncalibrated successor ML model to the ROC curve forthe calibrated successor model to ensure that is match or a substantialmatch between the ROC curves. A match between the ROC curves wouldindicate that the classification accuracy of the uncalibrated successorML model is preserved within the calibrated successor ML model.

4.1 Incumbent Machine Learning Model

S210, which includes collecting digital threat scores, functions tocollect sample digital threat scores generated by an incumbent digitalthreat machine learning (ML) model. Specifically, during a normal courseof operating the incumbent digital threat ML model, S210 may function tooperate a first threat score reservoir to collect samples of digitalthreat scores being generated at the incumbent digital threat ML model.Accordingly, S210 may function to collect sample digital threat scoresproduced in response to threat score requests (from service providers,customers, etc.) and/or threat scores that are automatically generatedby a system implementing the method 200.

In a preferred embodiment, the incumbent digital threat ML model may bea machine learning model that is specifically configured to evaluatedigital events (e.g., online transactions, online activities, etc.)and/or digital actors (users) and generate one or more of a threatclassification and a digital threat score based on the features of thedigital events and/or digital actors. Accordingly, a system (e.g.,system 100) implementing the incumbent digital threat ML model mayfunction to implement or operate a feature extractor that identifiesrelevant features of the digital events data, extract or collect thosefeatures from the digital events data, and provide the extractedfeatures as input machine learning input into the incumbent digitalthreat ML model. Thus, the system (or platform) may generate digitalthreat scores in accordance with the one or more embodiments describedin U.S. patent application Ser. No. 15/653,373, which is incorporated byreference in its entirety.

The digital threat score generation by the incumbent digital threat MLmodel may be synchronous or asynchronous. That is, in some embodiments,when a synchronous request for a threat score for a given event data oruser is received by a system implementing the method 200, the requesterwaits until the system responds directly to the request with the threatscore. Accordingly, a synchronous request may trigger an immediate(e.g., within 1 second to 2-3 minutes, etc.) or near-immediate threatscore generation and response by the system. Additionally, oralternatively, a system implementing the method 200 may asynchronouslygenerate a threat score and return the threat score to a client in theabsence of an explicit request for a threat score. Additionally, in thecase that an asynchronous request is received by the system, the systemmay function generate a threat score for a given event data or user at alater time and returns the generated threat score to a requestor with a(API) call back to the requestor at the later time. Thus, in anasynchronous mode of operating a system implementing the method 200 mayfunction to receive and process multiple asynchronous threat scorerequests and return threat scores in response to the requests as thethreat scores become available at a later time.

Additionally, or alternatively, S210 may function to specificallyconfigure the first digital threat score reservoir. As discussedbriefly, S210 may function to collect and record samples of digitalthreat scores generated by the incumbent digital threat ML model into afirst digital threat score reservoir. S210 may function to configure thefirst reservoir with logic that enables the first reservoir to arrangethe collected digital threat scores in a predetermined manner as well ascount the digital threat scores as they are collected. For instance, thefirst reservoir may include a counter that functions to increment by aninteger in an increasing manner for each digital threat score receivedand recorded within the first reservoir. In a preferred embodiment, S210may function to store a specific count (as metadata or the like) foreach specific digital threat score received at the first reservoir inassociation with the digital record for the specific digital threatscore. For instance, if a digital threat score of “80” is received atthe first reservoir and it is the third digital threat score collectedby the first reservoir, S210 may function to store the digital threatscore “80” in electronic association with the count of “3”. In this way,a correlation may be established between digital threat scores of theincumbent digital threat ML model and digital threat scores of asuccessor digital threat ML model.

It shall be noted that S210 may function to generate or collect anyadditional data associated with the collected digital threat scoresincluding a time range or period over which all the digital threatscores were collected, a time of capture or timestamp for each of thedigital threat scores, a first and last score time, a number of scorescaptures, and the like. These additional digital threat score data mayalso be stored in association with respective digital threat scores,preferably as metadata.

S210 may continue to collect digital threat scores produced by theincumbent digital threat ML model using the first digital threat scorereservoir until sufficient digital threat score values are collectedthat satisfy predetermined statistical values or thresholds.

It shall be noted that if multiple digital threat scoring models areimplemented for a specific service provider, S210 may function to storesamples of digital threat scores from each of those models andpreferably, store a model identifying value in association with eachdigital threat score collected in stored within a reservoir or the like.

S220, which includes generating a digital threat score distribution forthe incumbent machine learning model, functions to arrange the recordedsample digital threat scores collected from the incumbent digital threatML model into a score distribution.

Preferably, S220 functions to apply a cumulative distribution functionto the stored digital threat scores to generate a cumulativedistribution. In this way, the recorded digital threat scores may beillustrated in a score distribution in which the digital threat scoresare ranked in an ascending (or descending) manner and bounded betweenzero and one [0,1] (or [0, 100]). Thus, irrespective of the time atwhich the digital threat scores were generated by the digital threat MLmodel, the generated score distribution provides a comprehensible orderto the digital threat scores that enables a further capture ordetermination of quantiles of the score distribution.

Accordingly, once a score distribution of the stored digital threatscores is generated, S220 may additionally function to capture ordetermine quantiles for the score distribution of the digital threatscores produced by the incumbent digital threat ML model. The quantilesmay include a list of indices in which the indices representequally-spaced scores (or cumulative probabilities at equal points alongthe score distribution), and the score values at each index is theproportion of probability mass that is less than the score value of therespective index. Accordingly, the quantiles function to divide thescore distribution into equal parts. For instance, a score distributionmay be divided according to quartiles, percentiles, deciles, and thelike. It shall be noted that any suitable quantile for a given scoredistribution may be captured or determined.

Additionally, S220 may function to store identified quantiles of a givenscore distribution in association with the given score distribution.Specifically, S220 may function to store the identified quantiles in thefirst digital threat score reservoir in electronic association with theone or more score distributions of the incumbent digital threat ML modelstored therein.

4.2 Configuring a Successor Model

S230, which includes collecting digital threat scores of a successordigital threat machine learning (ML) model, functions to collect sampledigital threat scores generated by the successor digital threat MLmodel. The successor digital threat ML model may be an emergent scoringmodel that may function to succeed the incumbent digital threat ML modelfor producing digital threat scores for a specific service provider orthe like.

In some embodiments, S230 may function to operate the successor digitalthreat ML model in parallel or simultaneously with the incumbent digitalthreat ML model, however, the successor digital threat ML mode may beoperated in a shadow mode in which its scores are produced but notexposed (live) to a service provider or customer. In this regard, thesuccessor digital threat ML model may function to receive equivalent orsame digital event inputs as the incumbent digital threat ML model. Thatis, S230 may operate the successor digital threat ML model such thateach dataset (e.g., extracted features, metadata, etc.) for evaluationof a given digital event that is provided as input into the incumbentdigital threat ML model may also be provided to the successor digitalthreat ML model and preferably, provided as input in the same order. Inthis way, both the incumbent and the successor models are generatingcomparable digital threat scores. Comparable in the sense that thedigital threats scores produced by the incumbent and successor modelsfind basis in the same datasets without necessarily having a same scorevalue.

Additionally, or alternatively, S230 may function to provide a countvalue to each of the digital threat scores of the successor digitalthreat ML model. The count value may be stored (as metadata or the like)in association with a given digital threat score in the second digitalthreat score reservoir.

In operation, S230 may function to operate the successor digital threatML model in a private mode or a shadow mode in which the successordigital threat ML model receives digital events data (or extractedfeatures input, etc.) and generates digital threat scores therefrom.However, in the private or shadow mode, the digital threat scoresderived or generated by the successor digital threat ML model are notreturned or exposed to a service provider. Rather, S230 functions tocollect and store the generated digital threat scores produced by thesuccessor digital threat ML model in the private or shadow mode to asecond digital threat score reservoir as uncalibrated digital threatscores.

An uncalibrated digital threat score, as referred to herein maytypically refer to a digital threat score that is generated or producedby a (raw) successor digital threat ML model and that has not beencalibrated according to a reference model (e.g., incumbent model) or ascore distribution of a reference model. Accordingly, an uncalibrateddigital threat score may be a digital threat score that is in a raw formand/or untransformed condition relative to a digital threat score thatis calibrated or a digital threat score of a live incumbent model andpublicly returnable or exposable to a service provider as a valid oruseable digital threat score. In some embodiments, an uncalibrateddigital threat score may be used for improving a machine learning modeland general comprehension rather than as a score for indicating ameasure or likelihood of an actual digital threat to a service provideror the like based on the digital event data used to generate theuncalibrated digital threat score.

In a similar manner as described in S220, S240, which includesgenerating a digital threat score distribution for the successor digitalthreat ML model, functions to generate a score distribution using the(uncalibrated) digital threat scores of the successor digital threat MLmodel stored in the second digital threat score reservoir. That is, S240preferably applies a cumulative distribution function to the collectionof digital threat scores of the successor model that generates acumulative score distribution.

From the score distribution of the successor digital threat ML model,S240 may additionally function to identify and/or capture quantileswithin the score distribution and correspondingly, store the quantilesdata in electronic association with the score distribution from whichthe quantiles data was derived. In some embodiments, the quantiles maybe the same as the quantiles captured in the score distribution of thedigital threat scores produced by the incumbent digital threat ML model.

Alternatively, the quantiles of the score distribution of the successordigital threat ML model may be different from the quantiles captured inthe score distribution of the digital threat scores produced by theincumbent digital threat ML model. In latter embodiments, the quantilesof the successor model may be different from the incumbent model when ascore distribution of the successor model is sufficiently different froma score distribution of the incumbent model such that the proportion ofscores falling within a fixed score range under the successor modeldiffers meaningfully from the proportion of scores falling within thesame range under the incumbent model.

4.3 Configuring a Calibrated Successor Model

S250, which includes remapping the digital threat score distribution ofthe successor distribution machine learning model, functions tocalibrate the stored digital threat scores and/or the stored digitalthreat score distribution of the successor digital threat ML model. In apreferred embodiment, S250 may function to detect one or more remappingconditions that function to automatically trigger a remapping (orcalibration) of the uncalibrated digital threat scores of the successordigital threat ML model.

Generally, S250 functions to remap a score distribution of an incumbentdigital threat ML model to an uncalibrated score distribution of asuccessor digital threat ML model to generate a calibrated scoredistribution of the successor digital threat ML model based on quantilesdata captured from the score distribution of the incumbent model in amanner that preserves the score distribution of the incumbent modelwithin the score distribution of the successor model (i.e., quantilesremapping). Additionally, as discussed further below, the remappingfunction may also function to ensure that the classification accuracy ofthe uncalibrated successor model is preserved within the calibratedsuccessor model.

Additionally, it shall be noted that the remapping function, by virtueof being based on specific, timestamped threat score distributions,generally does not hold the calibrated threat score distribution of thesuccessor model fixed over time. However, at an outset of a transitionfrom an incumbent ML model to a successor ML model, S250 may implement afixed remapping function that causes the uncalibrated score distributionof the successor ML model to match the score distribution of theincumbent ML model over an initial, fixed period during which both theincumbent ML model and the successor ML model have recorded threatscores.

S250 may function to detect one or more remapping conditions includingan existence of a completed digital threat score distribution comprisingdigital threat scores generated by the incumbent digital threat MLmodel, an existence of quantiles of the score distribution of theincumbent digital threat ML model, and an existence of a completeddigital threat score distribution of the digital threat scores generatedby the successor digital threat ML model. Thus, S250 may function toscan each of the first digital threat score reservoir and the seconddigital threat score reservoir (or any other reservoir or database) toidentify or detect datasets or storage records corresponding to each ofthe required one or more remapping conditions.

In general terms, the one or more remapping conditions may be defined inany suitable manner and may not necessarily be tied to the scoredistributions and quantiles of an incumbent digital threat ML model.Rather, the one or more remapping conditions may be associated or tiedto any designated target digital threat score(s), threat scoredistributions, and/or machine learning model. Accordingly, S250 mayfunction to detect a score distribution and associated quantiles of anydesignated target model that may be necessarily different than anincumbent digital threat machine learning model that is actively used toproduce digital threat scores for a specific service provider of asystem and/or service implementing the method 200.

In one variant, S250 may function to trigger a remapping function basedon a partial score distribution of the incumbent digital threat ML model(e.g., a target model). That is, in some embodiments, a sufficientnumber of scores produced by the incumbent digital threat ML model maynot have been recorded into the first digital threat reservoir andtherefore, fails to meet a statistical threshold for generating acomplete score distribution. In such embodiments, even if thestatistical thresholds for completeness of the score distribution is notmet, S250 may generate the partially completed score distribution andcorrespondingly capture quantiles of the partial score distribution ofthe incumbent digital threat ML model and record the partial scoredistribution and quantiles within the first digital threat scorereservoir.

In this variant, a similar partial score distribution may be generatedfor a successor digital threat ML model and associated quantilescaptured therefrom and stored in a second digital threat scorereservoir. Since an implemented successor model may most likely use thesame digital event data inputs as used with the incumbent digital threatML model to produce digital threat scores, the second digital threatscore reservoir may most likely include a similar count digital threatscores as the first digital threat score reservoir that collects andrecords sample digital threat scores of the incumbent digital threat MLmodel.

In a further variant, if S250 detects that the one or more remappingconditions are not satisfied based on incomplete score distributions orthe like and a request or requirement for implementing a new model(e.g., a successor model) has been identified, S250 may function topredictively complete the score distributions for each of a target model(e.g., an incumbent model) and the successor model by deriving digitalevent data inputs for producing digital threat scores using the targetmodel and the successor model. That is, S250 may function to synthesizesufficient machine learning data inputs (e.g., synthetic digital eventsdata) for the incumbent model and the successor model that enablesdigital threat scores collected and stored therefrom by their respectivereservoirs to meet or satisfy the (completion) thresholds that indicatewhen a sufficient count of digital scores have been achieved forgenerating a complete score distribution.

In this further variant, S250 may function to derive or synthesizemachine learning data inputs using historical digital events data for aspecific service provider. Preferably, S250 functions to use theexisting digital events data that may have already been used as inputinto an incumbent digital threat ML model for a service provider asbasis to predict or derive synthetic or artificial digital events datainputs. Additionally, or alternatively, S250 may use platform data(e.g., data derived or collected from all computations of a digitalthreat mitigation platform, as described herein) or model data fromcomparable or cognate sources to assist in a calculation of syntheticdigital events data inputs for generating additional digital threatscores. It shall be noted that S250 may use any suitable source togenerate synthetic digital events or machine learning data inputs forthe target incumbent and successor models.

In yet an event further variant, S250 may function to switch theconfiguration or mode of the incumbent digital threat ML model and thesuccessor digital threat ML model from a first mode in which digitalevents data inputs (e.g., machine learning data inputs) are provided forproducing digital threat scores to a second mode in which no digitalevents data inputs are provided to the incumbent and successor modelsfor producing sufficient digital threat scores for completing scoredistributions. That is, in some embodiments, a target or incumbentdigital threat ML model as well as a successor model may be triggered tocomplete their score distributions by automatically generating digitalthreat scores event absent new digital events data. Accordingly, thesecond mode or the incumbent model and the successor model may includethe selective activation of predictive components of their respectivealgorithms that enable the respective models to generate synthetic orartificial digital threat scores without a need for digital events ormachine learning inputs. A synthetic or an artificial digital threatscore, as referred to herein, may typically include digital threatscores produced by a digital threat model that are produced withoutreceiving machine learning data inputs based on an occurrence of realworld digital events (e.g., an online fraud transaction, online contentabuse, etc.) and are self-generated by a target ML model and/orsuccessor ML model.

In response to detecting that one or more remapping conditions aresatisfied, S250 may additionally function to calibrate scoring outputsof the successor digital threat ML model. That is, S250 preferablyfunctions to identify, within a second digital threat score reservoir(or the like), a latest in time (uncalibrated) score distribution of asuccessor digital threat ML model and calibrate the digital threatscores of the uncalibrated score distribution based on a scoredistribution of a target model (e.g., an incumbent ML model).Preferably, the generation of the uncalibrated score distribution of thesuccessor ML model and the score distribution of the incumbent ML modelmay be based on same digital event data inputs. That is, for a givendigital event data input, each of the incumbent and successor ML modelsmay function to generate an output threat score.

Specifically, the detection of the satisfaction of the one or morepredicates to a remapping (i.e., the one or more remapping conditions)may function to trigger, a remapping function that generally operates toselect or identify a latest in time digital threat score distributionand associated quantiles data of an incumbent digital threat ML modeland a latest in time digital threat (uncalibrated) score distribution ofa successor model. In the remapping function, S250 may function to setthe score distribution of the incumbent ML model as a reference scoredistribution to which the uncalibrated score distribution of theincumbent ML model may be remapped based on. That is, S250 may generallyfunction to remap the digital threat scores within the uncalibratedscore distribution of the incumbent ML model with respect to digitalthreat scores within the score distribution of incumbent ML model.

In a preferred embodiment, the remapping function implemented by S250includes applying a monotonic function to the digital threat scores ofthe uncalibrated score distribution of the successor digital threat MLmodel. The monotonic function as applied by S250 may function to(approximately) preserve the score distribution of the incumbent modelsuch that the score distribution of the uncalibrated distribution of theincumbent model shares the score distribution of the incumbent model.

Additionally, as mentioned above, S250 may function to apply themonotonic remapping function in such a manner that preserves theaccuracy characteristics of the uncalibrated score distribution of thesuccessor digital threat ML model. That is, the monotonic remappingfunction as applied in S250 functions to preserve the receiver operatingcharacteristics (ROC) curve of the uncalibrated score distribution ofthe successor ML model within the ROC curve of the calibrated scoredistribution of the successor ML model. Accordingly, in preferredembodiments, S250 may function to apply the remapping function such thatthe ROC curves of the uncalibrated score distribution and the calibratedscore distribution of the successor ML model are the same orsubstantially the same (e.g., between 90% to 99%+ match or overlap ofarea under curve (AUC) the ROCs for the respective score distributions).

Additionally, or optionally, S255 may function to test a behavior of thecalibrated successor model to confirm that the classification accuracyof the calibrated successor model is improved relative to aclassification accuracy of the prior or incumbent model. Accordingly,S255 may function to enable same digital event data to be received bythe incumbent model as well as the calibrated successor model. S255 maysubsequently generate an ROC curve for each of the incumbent model andthe calibrated successor model to verify that the area under curve (AUC)of the ROC curve for the calibrated successor model is greater than theAUC of the ROC curve for the incumbent model. In the case that the AUCof the ROC curve of the calibrated successor model is greater than theAUC of the ROC curve of the incumbent model, S255 may function tovalidate the improved classification and/or predictive accuracy of thecalibrated successor model.

Additionally, or optionally, S255 may function to test a behavior of thecalibrated successor model relative to a classification accuracy of theuncalibrated successor model. Accordingly, S255 may function to enablesame digital event data to be received by the uncalibrated successormodel as well as the calibrated successor model. S255 may subsequentlygenerate an ROC curve for each of the uncalibrated successor model andthe calibrated successor model to verify that the area under curve (AUC)of the ROC curve for the calibrated successor model is the same as theAUC of the ROC curve for the uncalibrated successor model. In the casethat the AUC of the ROC curve of the calibrated successor model is thesame as the AUC of the ROC curve of the uncalibrated successor model,S255 may function to validate an equivalence of the uncalibrated andcalibrated scores of the successor model thereby indicating that theclassification and/or predictive accuracy of the uncalibrated successormodel is preserved within the calibrated successor model.

In such preferred embodiment, the remapping function implemented by S250may function to use the quantiles data stored in association with orelectronically linked to the score distribution of the incumbent MLmodel to generate quantiles for the uncalibrated score distribution ofthe successor ML model. Preferably, in remapping between the scoredistribution of the incumbent model and the uncalibrated scoredistribution of the successor model, S250 may function to write orproduce quantiles that are copied or substantially copied from theexisting quantiles of the score distribution of the incumbent model forgenerating a new calibrated score distribution for the successor model.Accordingly, S250 may first set the quantiles for the new calibratedscore distribution and subsequently, rewrite or convert the uncalibrateddigital threat scores of the uncalibrated score distribution of thesuccessor model into the indices defined by the quantiles of thecalibrated score distribution. The quantiles may include a plurality ofa set of values or thresholds that define indices into which a set ofdigital threat scores may reside therein. Example indices of quantilesmay include: 0.0-0.25 [1^(st) index], 0.25-0.50 [2^(nd) index],0.50-0.75 [3^(rd) index], and 0.75-1.0 [4^(th) index]. In this example,the quantiles comprise quartiles in which four equal indices areestablished.

S250 may function to apply the remapping function to create digitalthreat score pairings between each digital threat score of the scoredistribution of the uncalibrated successor model and each digital threatscore of the calibrated score distribution such that each uncalibrateddigital threat score of the uncalibrated score distribution is mapped toa respective digital threat score of the calibrated score distributionof the successor digital threat ML model. Preferably, the uncalibrateddigital threat scores are mapped to calibrated digital threat scoresappearing in the same rank order or count within the calibrated scoredistribution of the successor digital threat ML model. For instance, twothreat scores from the uncalibrated successor model may be mapped to twocalibrated scores of the successor model that are in the same rankorder. Accordingly, in such example, if a first rank ordereduncalibrated score is smaller than a second rank ordered uncalibratedscore of the uncalibrated successor model, then the remapping functionoperates such that a first calibrated score will also be smaller than asecond calibrated score of the calibrated successor model. Accordingly,the remapping function is preferably rank order preserving such that theorder of the uncalibrated scores of the uncalibrated successor model ispreserved within the calibrated scores of the calibrated successormodel.

Accordingly, once the uncalibrated digital threat scores of theuncalibrated score distribution of the successor model are mapped toequally ranked or ordered digital threat scores of the scoredistribution of the calibrated successor model and the quantiles for thenew calibrated score distribution are set, S250 applying the monotonicfunction rewrites the uncalibrated scores to fit within the quantiles ofthe new calibrated score distribution such that a count of digitalthreat scores set between each index of the new calibrated scoredistribution matches (or substantially matches) a count of digitalthreat scores of the score distribution of the incumbent model. That is,the new calibrated score distribution of the successor model may beremapped or written such that the new calibrated score distributioncaptures the same (or substantially the same) quantiles as the scoredistribution of the incumbent model and captures a same (orsubstantially the same) count of digital threat scores within theindices of the quantiles of the new calibrated score distribution as thecount of digital threat scores within the indices of the quantiles ofthe score distribution of the incumbent model.

It shall be noted that the remapping function applying a monotonicfunction to the uncalibrated digital threat scores of the uncalibrateddigital threat score distribution functions to rewrite one or more ofthe uncalibrated digital threat scores such that the rewritten orcalibrated digital threat scores appear in the same rank order andwithin a same index as it paired digital threat score within the scoredistribution of the incumbent model.

Accordingly, the remapping between the score distribution of theincumbent model and the score distribution of the uncalibrated scoredistribution of the uncalibrated successor model generates a newcalibrated score distribution of the successor model in which the newcalibrated score distribution of the successor model approximatelymatches the score distribution of the incumbent model thereby maintainscore distributions between incumbent model and the calibrated successormodel. It shall be noted that while the uncalibrated scores may berewritten or transformed to maintain a rank order position and indexposition as its pairing within the score distribution of the incumbentmodel, the remapping function may not necessarily rewrite the calibratedscore to match its corresponding pair within the score distribution ofthe incumbent model. Rather, the monotonic function of the remappingaims to preserve in the new calibrated score distribution the rank orderand index location of the pairings of the digital threat scores of thescore distribution of the incumbent model and the digital threat scoresof the uncalibrated score distribution of the successor model.

Additionally, or alternatively, once a remapping of the uncalibrateddigital threat score distribution to a new calibrated digital threatscore distribution is established, S250 may function to record or storethe new calibrated score distribution within a third reservoir. The newcalibrated score distribution may be stored within the third reservoirin electronic association with the score distribution of the successormodel and the uncalibrated score distribution of the incumbent modelused in producing the new calibrated score distribution.

S260, which includes returning one or more digital threat scores,functions to identify a remapping status of the third reservoir todetermine whether a remapping is recorded therein and return a digitalthreat score based on a remapping status. Specifically, while asuccessor model is implemented in a private or shadow mode and aremapping between the digital threat score distributions of theincumbent model and the successor model is not complete, S260 typicallyfunctions to return or publicly expose a digital threat score, inresponse to a request for a digital threat score or the like, basedprimarily on scores generated by the incumbent model. That is, in theabsence of a valid or complete remapping and production of a calibratedscore distribution of the successor model, S260 continues to publiclyexpose or return only those digital threat scores produced by incumbentmodels.

If the remapping status within the third reservoir indicates that aremapped or calibrated digital threat score distribution is recordedtherein, S260 may function to publicly expose or return the calibrateddigital threat score produced by the successor digital threat ML modelin the place of the corresponding digital threat score generated by theincumbent digital threat ML model. That is, when the remappingconditions are satisfied, a system and/or platform implementing themethod 200 produces the calibrated digital threat scores in response toa request for a digital threat score.

In one variant, once the remapping conditions are satisfied (e.g., acalibrated score distribution is recorded in the 3^(rd) reservoir), S260may function to return both the calibrated digital threat score of thesuccessor model as well as the digital threat score of the incumbentmodel. A system or platform implementing the method 200 may function topresent both the calibrated digital threat score and the incumbentdigital threat score for a predetermined period and once the period iscompleted, S260 may return only the calibrated digital threat score.

In a second variant, S260 may function to return an ensemble of thethreat scores produced by the calibrated digital threat score of thesuccessor model and the threat score of the incumbent model. In suchvariant, S260 may function to compute a simple average between thecalibrated threat score and the threat score of the incumbent model.Additionally, or alternatively, S260 may function to perform a weightedaverage in which different coefficients are applied against thecalibrated score and the threat score of the incumbent model. In apreferred embodiment of this second variant, S260 may function to applya coefficient or weight to the calibrated threat score that is largerthan the coefficient or weight applied to the threat score of theincumbent model.

The system and methods of the preferred embodiment and variationsthereof can be embodied and/or implemented at least in part as a machineconfigured to receive a computer-readable medium storingcomputer-readable instructions. The instructions are preferably executedby computer-executable components preferably integrated with the systemand one or more portions of the processors and/or the controllers. Thecomputer-readable medium can be stored on any suitable computer-readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a general or applicationspecific processor, but any suitable dedicated hardware orhardware/firmware combination device can alternatively or additionallyexecute the instructions.

Although omitted for conciseness, the preferred embodiments includeevery combination and permutation of the implementations of the systemsand methods described herein.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

What is claimed:
 1. An online system for evolving one or more machinelearning models for identifying new and/or different digital threats andneutralizing digital threats by calibrating the one or more machinelearning model, the system comprising: a non-transitorycomputer-executable medium storing computer instructions that whenexecuted by one or more computer processors provides an applicationprogramming interface that is in interactive communication with one ormore endpoints of a machine learning service to control and/or operateone or more functions of an ensemble of machine learning models that areused to generate a digital threat score based on digital event data,wherein the digital event data relate to online activities of one ormore users involved with one or more digital services provided by anonline service provider; the machine learning service being implementedby one or more hardware computer servers comprises: a digital threatscore reservoir that: collects incumbent digital threat scores generatedby an incumbent machine learning model and successor digital threatscores generated by an uncalibrated successor digital threat machinelearning (ML) model; generates an incumbent threat score distributionbased on the incumbent digital threat scores and generates anuncalibrated successor threat score distribution based on the successordigital threat scores; captures quantiles data from the incumbentdigital threat score distribution and the uncalibrated successor scoredistribution; a remapping module that generates a calibrated successordigital threat machine learning model by: applying the quantiles of theincumbent digital threat score distribution to the uncalibratedsuccessor digital threat score distribution; remapping the successordigital threat scores of the successor digital threat score distributionbased on the incumbent digital threat scores of the incumbent digitalthreat score distribution; using the remapping of the successor digitalthreat scores and the quantiles of the incumbent digital threat scoresto transform the successor digital threat scores to calibrated digitalthreat scores of the calibrated digital threat score distribution; andwherein response to transforming the successor digital threat scores tocalibrated successor digital threat scores, returning a calibratedsuccessor digital threat score in response to the request for thedigital threat score for a digital event or a digital actor; wherein inresponse to returning the calibrated successor digital threat score,performing by the online service provider one or more of automaticallyapproving, holding, and cancelling an activity or an online transactionassociated with the digital event data based on a comparison of thecalibrated digital threat score to an online service provider-specificscore threshold to neutralize a digital threat associated with theactivity or the online transaction.
 2. The system according to claim 1,wherein: in response to receiving a request for a digital threat scorefor a digital event or for a digital actor, the machine learning serviceprovides equivalent digital event data as machine learning input intoboth the incumbent digital threat ML model and the successor digitalthreat ML model.
 3. The system according to claim 1, wherein thecalibrating includes: generating a monotonic function based on thequantiles data of the incumbent digital threat score distribution andthe quantiles data of the uncalibrated successor digital threat scoredistribution; applying the monotonic function to the uncalibratedsuccessor digital threat score distribution to preserve a scoredistribution pattern of the incumbent digital threat score distribution.4. The system according to claim 1, wherein: the remapping is triggeredbased on a detection of an existence of a record of the quantiles dataof the incumbent digital threat score distribution and an existence ofrecords of the incumbent digital threat score distribution and thesuccessor digital threat score distribution.
 5. A method of evolving oneor more machine learning models of a machine learning-based digitalthreat detection system for identifying new and/or different digitalthreats and neutralizing digital threats by calibrating the one or moremachine learning models, the method comprising: implementing anon-transitory computer-executable medium storing computer instructionsthat when executed by one or more computer processors provides anapplication programming interface that is in interactive communicationwith one or more endpoints of a machine learning service to controland/or operate one or more functions of an ensemble of machine learningmodels that are used to generate a digital threat score based on digitalevent data, wherein the digital event data relate to online activitiesof one or more users involved with one or more digital services providedby an online service provider; collecting digital threat scores of anincumbent digital threat machine learning (ML) model; identifying anincumbent digital threat score distribution of the digital threat scoresof the incumbent digital threat ML model; identifying quantiles data ofthe incumbent digital threat score distribution; collecting digitalthreat scores of a successor digital threat machine learning (ML) model;identifying an uncalibrated successor digital threat score distributionof the digital threat scores of the successor digital threat ML model;identifying quantiles data of the uncalibrated successor digital threatscore distribution; calibrating the uncalibrated successor digitalthreat ML model by calibrating the digital threat scores of thesuccessor digital threat score distribution based on the quantiles dataof the incumbent digital threat score distribution and the incumbentdigital threat score distribution, wherein the calibrating includes:remapping the digital threat scores of the uncalibrated successordigital threat score distribution based on the digital threat scores ofthe incumbent digital threat score distribution, wherein the remappingincludes: applying the quantiles data to the remapping of the digitalthreat scores of the uncalibrated successor digital threat scoredistribution; configuring the digital threat scores of the uncalibratedsuccessor digital threat score distribution to fit within a plurality ofindices of the quantiles data; wherein response to the remapping of thedigital threat scores of the uncalibrated successor digital threat scoredistribution, the machine learning system publishes calibrated successordigital scores in lieu of the incumbent digital threat scores based onone or more requests for digital threat scores; and wherein in responseto obtaining a calibrated successor digital threat score, performing bythe online service provider one or more of automatically approving,holding, and cancelling an activity or an online transaction associatedwith the digital event data based on a comparison of the calibrateddigital threat score to an online service provider-specific scorethreshold to neutralize a digital threat associated with the activity orthe online transaction.
 6. The method according to claim 5, furthercomprising: implementing the incumbent digital threat ML model in apublic mode of operation, wherein the digital threat scores generated bythe incumbent ML model are publicly returned in response to a requestfor a digital threat score for a digital event or a digital actor, andimplementing the uncalibrated successor digital threat ML model in aprivate mode of operation, wherein the digital threat scores generatedby the uncalibrated successor digital threat ML model are privatelystored and not publicly returned in response to the one or more requestsfor the digital threat score for the digital event.
 7. The methodaccording to claim 6, wherein: implementing the incumbent digital threatML model and the successor digital threat ML model are operated inparallel.
 8. The method according to claim 5, further comprising: inresponse to receiving a request for a digital threat score for a digitalevent or a digital actor, providing equivalent digital event dataassociated with the digital event or the digital actor as machinelearning input into both the incumbent digital threat ML model and theuncalibrated successor digital threat ML model.
 9. The method accordingto claim 5, wherein the remapping includes: generating a monotonicfunction based on the quantiles data of the incumbent digital threatscore distribution and the quantiles data of the uncalibrated successordigital threat score distribution; applying the monotonic function tothe uncalibrated successor digital threat score distribution to preservethe incumbent digital threat score distribution.
 10. The methodaccording to claim 5, wherein: the remapping is triggered based on adetection of a record of the quantiles data of the incumbent digitalthreat score distribution and records of the incumbent and successordigital threat score distributions.
 11. The method according to claim 5,further comprising: generating a count value for each of the incumbentdigital threat scores and for each of the successor digital threatscores, wherein: the remapping includes pairing incumbent digital threatscores and successor digital threat scores having a same count value.12. The method of claim 5, wherein the machine learning servicecalibrates the uncalibrated successor machine learning model withoutknowledge of automated decisioning thresholds of a service provider. 13.The method of claim 5, further comprising: testing a classificationaccuracy of the calibrated successor machine learning model, whereintesting includes: providing a same digital event data input set to theuncalibrated successor machine learning model and the calibratedsuccessor machine learning model; generating score distributions of eachof the uncalibrated successor machine learning model and the calibratedsuccessor machine learning model; generating a receiver operatingcharacteristics curve for the score distributions of each of theuncalibrated successor machine learning model and the calibratedsuccessor machine learning model; identifying a level of match betweenthe receiver operating characteristics curves for the uncalibratedsuccessor machine learning model and the calibrated successor machinelearning model.
 14. The method of claim 5, further comprising: measuringa sensitivity level of a service provider to a proposed migration to theuncalibrated successor model; and if the sensitivity level satisfies oneor more sensitivity thresholds, triggering an automatic calibration ofthe uncalibrated successor machine learning model.
 15. The method ofclaim 5, further comprising: measuring a sensitivity level of a serviceprovider to a proposed migration to the uncalibrated successor model;and if the sensitivity level does not satisfy one or more sensitivitythresholds, triggering an automatic live implementation of theuncalibrated successor machine learning model.
 16. A method of evolvingone or more machine learning models of a machine learning-based digitalthreat detection system for identifying new and/or different digitalthreats and neutralizing digital threats by calibrating the one or morea machine learning models, the method comprising: implementing anon-transitory computer-executable medium storing computer instructionsthat when executed by one or more computer processors provides anapplication programming interface; receiving, via the applicationprogramming interface, one or more requests by an online serviceprovider for one or more machine learning scores; generating a firstscore distribution for an active machine learning model and a secondscore distribution for a nascent machine learning model based on a sameinput dataset; acquiring quantiles data from each of the first andsecond score distributions; computing a monotonic remapping functionbased on the quantiles data for each of the first and the second scoredistributions; transforming the nascent machine learning model to acalibrated machine learning model by applying the monotonic remappingfunction to the second score distribution of the nascent machinelearning model; implementing the calibrated machine learning modelwithin a machine learning service and returning the one or more machinelearning scores using the calibrated machine model in response to theone or more requests for machine learning scores; and wherein inresponse to obtaining the one or more machine learning scores,performing by the online service provider one or more of automaticallyapproving, holding, and cancelling an activity or an online transactionassociated with the digital event data based on a comparison of the oneor more machine learning score to an online service provider-specificscore threshold to neutralize a digital threat associated with theactivity or the online transaction.